Method for facilitating world wide web searches utilizing a document distribution fusion strategy

ABSTRACT

A computer-implemented method for facilitating World Wide Web Searches and like database searches by combining search result documents, as provided by separate search engines in response to a query, into one single integrated list so as to produce a single document with a ranked list of pages, by forming a set of selected queries, the queries including respective terms, for which selected queries relevance data from past data is known, herein referred to as training queries, in a vector space comprising all training queries, the relevance data comprising judgments by a user as to whether a page is appropriate for a query which retrieved it. Further steps in the method are identifying a set of k most similar training queries to current query q, computing an average relevant document distribution of the k queries within the training queries&#39; search results for each of the search engines, using the computed relevant document distributions, finding an optimal number of pages to select from the result set of each search engine when N total pages are to be retrieved, and creating a final retrieved set by forming the union of the top λ s  pages from each search engine.

The present invention relates to an automatic method for facilitatingWorld Wide Web Searches and, more specifically, to an automatic methodfor facilitating World Wide Web Searches by exploiting the differencesin the search results of multiple search engines to produce a singlelist that is more accurate than any of the individual lists from whichit is built.

Text retrieval systems accept a statement of information need in theform of a query, assign retrieval status values to documents in thecollection based on how well the documents match the query, and return aranked list of the documents ordered by retrieval status value. Datafusion methods that combine the search results of different queriesrepresenting a single information need to produce a final ranking thatis more effective than the component rankings are well-known. SeeBartell, B. T., Cottrell, G. W., and Belew, R. K.: Automatic combinationof multiple ranked retrieval systems; Proceedings of SIGIR-94; July,1994. Belkin, N. J. et al.: The effect of multiple query representationson information system performance; Proceedings of SIGIR-93; June, 1993.Fox, E. A. and Shaw, J. A. Combination of multiple searches. Proceedingsof TREC-2; March, 1994.

However, these fusion methods determine the rank of a document in thefinal list by computing a function of the retrieval status values ofthat document in each of the component searches. The methods aretherefore not applicable when the component searches return only theordered list of documents and not the individual status values.

The World Wide Web is a collection of information-bearing units called"pages" interconnected by a set of links. To help users find pages ontopics that are of interest to them, several groups provide searchengines that accept a statement of user need (in either English or amore formal query language) and return a list of pages that match thequery. A list is usually ordered by a similarity measure computedbetween the query and the pages. While each of the search engines inprinciple searches over the same set of pages (the entire Web), the sizeof the Web and the imprecise nature of the search algorithms frequentlycauses different search engines to return different lists of pages forthe same query.

Search engines such as Excite and Alta Vista provide a query interfaceto the information in these pages, and, like traditional text retrievalsystems, return a ranked list of pages ordered by the similarity of thepage to the query. See Steinberg, Steve G.: Seek and Ye Shall Find(Maybe); Wired; May, 1996. Because the search engines process queries indifferent ways, and because their coverage of the Web differs, the samequery statement given to different engines often produces differentresults. Submitting the same query to multiple search engines, forexample such as Quarterdeck's WebCompass product does, can improveoverall search effectiveness. See QuarterDeck. URL:http://arachnid.qdeck.com/qdeck/products/webcompass.

In accordance with an aspect of the invention, a method provides forcombining the results of the separate search engines into a singleintegrated ranked list of pages in response to a query. UnlikeWebCompass, the method does not keep the search results separated by thesearch engine that produced the result, but forms a single ranked list.Unlike the traditional fusion methods, the method in accordance with theinvention can produce a single ranking despite the fact that most searchengines do not return the similarities that are computed for individualpages.

The method in accordance with the invention utilizes a particularapplication of algorithms developed to combine the results of searcheson potentially disjoint databases. See Towell, G., et al.: LearningCollection Fusion Strategies for Information Retrieval; Proceedings ofthe 12th Annual Machine Learning Conference; July, 1995. Voorhees, E.M., Gupta, N. K., and Johnson-Laird, B.: The Collection Fusion Problem;Proceedings of TREC-3, NIST Special Publication 500-225; April, 1995;pp. 95-104. Voorhees, E. M., Gupta, N. K., and Johnson-Laird, B.:Learning Collection Fusion Strategies; Proceedings of SIGIR-95; July,1995; pp. 172-179.

In accordance with another aspect of the invention, acomputer-implemented method for facilitating World Wide Web Searches bycombining search result documents, as provided by separate searchengines in response to a query, into one single integrated list so as toproduce a single document with a ranked list of pages, the methodcomprises the steps of (a) forming a set of selected queries, thequeries including respective terms, for which selected queries relevancedata from past data is known, herein referred to as training queries, ina vector space comprising all training queries, the relevance datacomprising judgments by a user as to whether a page is appropriate for aquery which retrieved it; (b) identifying a set of k most similartraining queries to current query q; (c) computing an average relevantdocument distribution of the k queries within the training queries'search results for each of the search engines; (d) using the computedrelevant document distributions, finding an optimal number of pages toselect from the result set of each search engine then N total pages areto be retrieved; and (e) creating a final retrieved set by forming theunion of the top λ_(s) pages from each search engine.

An object of the present invention is to approximate the effectivenessof a single text retrieval system despite the collection beingphysically separated. Another object of the present invention is tocombine the results of multiple searches of essentially the samedatabase so as to improve the performance over any single search.

In accordance with another aspect of the invention, the present methodfor facilitating World Wide Web searches utilizing a query clusteringfusion strategy uses relevance data - - - judgments by the user as towhether a page is appropriate for the query which retrieved it - - -from past queries to compute the number of pages to select from eachsearch engine for the current query. In the present description, the setof queries for which relevance data is known is called the trainingqueries. The terms "page" and "document" are used interchangeably.

FIGS. 1 and 2 show flow charts helpful to a fuller understanding of theinvention.

The function F_(s) ^(q) (N), called a relevant document distribution,returns the number of relevant pages retrieved by search engine s forquery q in the ranked list of size N. The fusion method in accordancewith the present invention, builds an explicit model of the relevantdocument distribution of the joint search. The model is created bycomputing the average relevant document distribution of the k nearestneighbors of the current query, q. The nearest neighbors of q are thetraining queries that have the highest similarity with q.

To compute query-query similarities, the present invention utilizes avector representation of the queries. The vector queries are created byremoving a set of high-frequency function words such as prepositionsfrom the query text, stemming the remaining words (i.e., removingsuffixes to conflate related words to a common root), and assigning aweight to each term equal to the number of times the term occurs in thetext (term frequency weights). The cosine of the angle between two queryvectors is used as the queries'similarity.

The average relevant document distribution over k queries is computed bytaking the average of the number of relevant documents retrieved by theset of queries after each document retrieved. Once the average relevantdocument distribution is computed for the current query for each searchengine, the distributions and the total number of documents to beretrieved are passed to a maximization procedure. This procedure findsthe cut-off level for each search engine that maximizes the number ofrelevant documents retrieved (the current maximization procedure simplydoes an exhaustive search). The computed cut-off levels are the numberof documents selected from the result set of each search engine. Thesteps of the fusion process in accordance with the invention areessentially as follows.

A. Find the k most similar training queries to current query q

1. Using standard techniques, create query vectors in a vector spaceconsisting of all training queries. Weight terms in queries using afunction that is proportional to the number of times the term occurs inthe query.

2. Create a query vector for the current query in the same vector space.Compute a vector similarity measure between the current query and alltraining queries.

3. Select the k training queries with the highest similarities.

B. Within the training queries' search results for each search engine,compute the average relevant document distribution of the k queries.

1. A relevant document distribution for a query q gives for each rank rthe number of relevant documents retrieved at or below rank r by queryq. The average distribution over a set of queries gives the mean numberof relevant documents retrieved at or below rank r over the query set.

C. Using the computed relevant document distributions, find the optimalnumber of pages to select from the result set of each search engine whenN total pages are to be retrieved.

1. Using any optimization technique (we use brute force), compute thenumber of pages that should be retrieved from each search engine (λ_(s))such that the total number of pages retrieved is N and the maximumpossible number of relevant pages is retrieved subject to the constraintthat e.g., to retrieve the page at rank 5 from a collection pages atranks 1--4 must also be retrieved.

2. There may be different combinations of pages retrieved from thesearch engine results that retrieve the maximum possible number ofrelevant pages. Choose any one of the combinations. Distribute "spill",the number of pages that can be retrieved from any search engine withoutaffecting the number of relevant retrieved, in proportion to the numberof pages that would otherwise be retrieved from that collection.

D. Create the final retrieved set by forming the union of the top λ_(s)pages from each search engine.

1. Rank pages in the final retrieved set probabilistically using abiased c-faced die.

(a) To select the page to be in the next rank r of the final ranking,roll a c-faced die that is biased by the number of pages remaining to beplaced in the final ranking from each of the search engines. Select thesearch engine whose number corresponds to the die roll and place thenext page from that engine's ranking into the final ranking.

(b) Repeat until all N pages have been placed in the final ranking.

The parameter k is used to control the amount of generalization madefrom the training queries. Too few queries cause the predicted relevantdocument distribution to be too specific to the training queries, whiletoo many queries cause different topic areas to be mixed resulting intoo generic of a distribution.

As used herein, a roll of an unbiased c-faced die selects a number inthe range from 1 to c with a uniform probability of 1/c; however, inorder to produce the final ranking, it is desired to bias theprobability of selecting a search engine, numbered from 1 to c, by thenumber of pages it has to place in the ranking. This means that the pageplace in the first rank will, with higher probability, be selected fromthe search engine that contributed the most pages to the retrieved set.As pages are placed in the final ranking, the search engine with themost pages remaining to be placed will change, and thus the specificprobabilities of selecting a search engine also change.

A fusion method for facilitating World Wide Web Searches utilizing aquery clustering fusion strategy is disclosed in a copending patentapplication by the present Inventor, entitled Method for facilitatingWorld Wide Web Searches Utilizing a Query Clustering Fusion Strategy andfiled on even date herewith and whereof the disclosure is hereinincorporated by reference to the extent it is not incompatible with thepresent invention. As therein disclosed, the fusion method does notattempt to form an explicit model of a search engine's relevant documentdistribution.

Instead, that system learns a measure of the quality of a search for aparticular topic area by that engine. The number of pages selected froman engine for a new query is proportional to the value of the qualitymeasure computed for that query. As in the approach disclosed in theabove-referenced application, the fusion strategy in accordance with theinvention uses query vectors. Topic areas are represented as centroidsof query clusters. For each search engine, the set of training queriesis clustered using the number of (relevant and irrelevant) documentsretrieved in common between two queries as a similarity measure. Theassumption is that if two queries retrieve many documents in common theyare about the same topic. The centroid of a query cluster is created byaveraging the vectors of the queries contained within the cluster. Thiscentroid is the system's representation of the topic covered by thatquery cluster.

The training phase also assigns to a cluster a weight that reflects howeffective queries in the cluster are for that search engine - - - thelarger the weight, the more effective the queries are believed to be.The average number of relevant pages retrieved by queries in the clusteris used as a cluster's weight.

After training, queries are processed as follows. The cluster whosecentroid vector is most similar to the query vector is selected for thequery and the associated weight is returned. The set of weights returnedover all the search engines is used to apportion the final retrieved setsuch that when N pages are to be returned and w_(s) is the weightreturned by engine s, (w_(s) /Σw_(k))*N (rounded appropriately)documents are selected from engine s. For example, assume the totalnumber of pages to be retrieved is 100, and there are five searchengines.

If the weights returned by the engines are 4, 3, 3, 0, 2, then the first33 pages returned by engine 1 would be selected, the first 25 pages fromeach of engines 2 and 3 would be selected, no pages would be selectedfrom engine 4, and the first 17 pages from engine 5 would be selected.However, if the weights returned were 4, 8, 4, 0, 0 then 25 pages wouldbe selected from each of engines 1 and 3, and 50 pages would be selectedfrom engine 2. The weight of a cluster for a single engine in isolationis not meaningful; it is the relative difference in weights returned bythe set of search engines over which the fusion is to be performed thatis important. Of course, many variations of this scheme, such as forcingsmall weights to zero when the difference between weights is very large,are also possible.

The current implementation uses the Ward clustering method and thereciprocal of the number of documents retrieved in common in the top 100pages as the distance metric to cluster the training queries. A singleset of clusters is produced from the resultant dendogram by cutting thedendogram at a pre-determined distance. The weight assigned to eachcluster is the average number of relevant documents in the top L ranks.The similarity between a cluster centroid and a query is computed as thecosine of the two vectors, where each vector uses term frequencyweights.

In the query clustering fusion strategy, the parameter L controls partof the generalization made from the training queries. The number ofdocuments used to compute query-query similarities for the clusteringroutine will also have an effect. Steps of the method disclosed in theabove-referenced copending patent application can be summarized asfollows.

A. Train for each search engine:

1. Cluster training queries and build cluster centroids.

(a) Apply Ward's clustering algorithm, using the number of pagesretrieved in common at a rank less than or equal to a parameter L as thesimilarity between two queries.

(b) Form clusters from hierarchy by considering all queries that clusterabove a certain threshold to belong to the same cluster.

(c) Form centroid for a particular cluster by creating the mean vectorover all query vectors in the cluster.

i. Create query vectors from query text using standard vector processingtechniques; weight the terms using a function that is proportional tothe number of times the term occurs in the query.

ii. The weight of a term in the centroid vector is the sum of itsweights in the vectors of the queries in the cluster divided by thenumber of queries in the cluster.

2. Assign weights to each cluster reflecting the number of relevantpages expected to be obtained by this search engine for queries similarto those in the cluster.

(a) Compute a cluster's weight as the mean number of relevant pagesretrieved at a rank less than or equal to a parameter L over all thequeries in the cluster.

B. To process an incoming query, for each search engine,

1. Find the cluster centroid that is most similar to the query.

(a) Create a query vector for the current query in the vector space ofthe training queries.

(b) Compute a vector similarity measure (e.g., the cosine) between thecurrent query vector and each of the centroids.

(c) Choose the centroid that has the greatest similarity.

2. Return the weight associated with the selected cluster as the weightof the current search engine.

C. Apportion the N slots in the retrieved set according to the weightsreturned by each search engine.

1. Sum the weights returned by the set of engines.

2. Select the top weight-of-this-engine/sum (rounded down) pages fromthe retrieved set of each engine.

3. When fewer then N pages are retrieved due to rounding, select 1 morepage from the most highly weighted engines until N pages are retrieved.(Break ties arbitrarily.)

4. Rank pages in the retrieved set probabilistically using a biasedc-faced die.

(a) To select the document to be in the next rank r of the finalranking, roll a c-faced die that is biased by the number of pagesremaining to be placed in the final ranking from each of the engines.Select the engine whose number corresponds to the die roll and place thenext page from that engine's ranking into the final ranking.

(b) Repeat until all N pages have been placed in the final ranking.

The invention has been described by way of an exemplary embodiment.Various changes and modifications will be apparent to one skilled in theart to which it pertains. While reference has been made to the WorldWide Web in conjunction with searches, it is intended and should beunderstood that what is herein intended is a data base as represented bythe World Wide Web, of that type and not necessarily so named. Suchchanges and modifications are intended to be within the spirit and scopeof the invention which is defined by the claims following.

We claim:
 1. A computer-implemented method for facilitating World WideWeb Searches and like searches by combining search result documents, asprovided by separate search engines in response to a query, into onesingle integrated list so as to produce a ranked list of pages, saidmethod comprising the steps of:(a) forming a set of selected queries,said queries including respective terms, for which selected queriesrelevance data from past data is known, herein referred to as trainingqueries, in a vector space comprising all training queries, saidrelevance data comprising judgments by a user as to whether a page isappropriate for a query which retrieved it; (b) identifying a set of kmost similar training queries to current query q; (c) computing anaverage relevant document distribution of said k queries within saidtraining queries' search results for each of said search engines; (d)using said computed relevant document distributions, finding an optimalnumber of pages to select from the result set of each search engine whenN total pages are to be retrieved; and (e) creating a final retrievedset by forming the union of the top λ_(s) pages from each search engine.2. A computer-implemented method in accordance with claim 1, whereinstep (a) comprises a step of(A) weighting each given term in saidselected queries using a function that is proportional to that number oftimes said given term occurs in the query.
 3. A computer-implementedmethod in accordance with claim 2, wherein step (a) comprises:(B)creating a query vector for a current query in said vector space; and(C) computing a vector similarity measure between said current query andall training queries.
 4. A computer-implemented method in accordancewith claim 1, wherein step (a) comprises(C) selecting those k trainingqueries with the highest similarities.
 5. A computer-implemented methodin accordance with claim 1, wherein in step (c), said average relevantdocument distribution is such that a relevant document distribution fora query q gives for each rank r the number of relevant documentsretrieved at or below rank r by query q and the average distributionover a set of queries gives the mean number of relevant documentsretrieved at or below rank r over the query set.
 6. Acomputer-implemented method in accordance with claim 1, wherein step (d)comprises:(D) computing, by using a selected optimization technique, anumber of pages that should be retrieved from each respective searchengine such that a total number of pages N is retrieved and a maximumpossible number of relevant pages is retrieved subject to apredetermined constraint.
 7. A computer-implemented method in accordancewith claim 6, wherein said predetermined constraint is typified by aconstraint such that to retrieve a page at rank R from a collection,pages at rank 1 through rank (R-1) must also be retrieved.
 8. Acomputer-implemented method in accordance with claim 6, wherein, if instep (D) there be different combinations of pages retrieved from thosesearch engine results that retrieve a maximum possible number ofrelevant pages, then select any one of said different combinations.
 9. Acomputer-implemented method in accordance with claim 8, wherein step (A)comprises(E) distributing a number of pages that can be retrieved fromany of said search engines without affecting the number of relevantdocuments retrieved, in proportion to that number of pages that wouldotherwise be retrieved from that collection.
 10. A computer-implementedmethod in accordance with claim 9, wherein step (E) comprises rankingpages in said final retrieved set probabilistically in accordance with arule of a die roll using a biased c-faced die.
 11. Acomputer-implemented method in accordance with claim 10, wherein, toselect the page to be in the next rank r of the final ranking, step (E)comprises rolling a c-faced die that is biased by the number of pagesremaining to be placed in said final ranking from each of said searchengines.
 12. A computer-implemented method in accordance with claim 11,wherein step (A) comprises selecting that search engine whose numbercorresponds to said die roll; andplacing the next page from thatengine's ranking into said final ranking.
 13. A computer-implementedmethod in accordance with claim 12, wherein the steps of claim 12 arerepeated until all N pages have been placed in said final ranking.
 14. Acomputer-implemented method for facilitating World Wide Web Searches andlike searches by combining search result documents from a joint search,as provided by separate search engines in response to a query, into onesingle integrated list so as to produce a ranked list of pages, saidmethod comprising the steps of:(a) forming a relevant documentdistribution, in accordance with a function F_(s) ^(q) (N) which returnsthat number of relevant pages retrieved by search engine s for a currentquery q in a ranked list of size N; (b) forming an explicit model ofsaid relevant document distribution of said joint search by computing anaverage relevant document distribution of the k nearest neighbors ofsaid current query, q, said nearest neighbors of q being those trainingqueries that have highest similarity with q, said training queries beinga set of selected queries, including respective terms, for whichselected queries relevance data from past data is known, said relevancedata comprising judgments by a user as to whether a page is appropriatefor a query which retrieved it; (c) computing query-query similarities,by generating a vector query representation of said queries whereinvector queries are created by removing a set of high-frequency functionwords, such as prepositions from query text, stemming remaining words,that is, removing suffixes to conflate related words to a common root,and assigning a term frequency weight to each term equal to the numberof times the term occurs in said text; (d) utilizing the cosine of theangle between two query vectors as the queries' similarity; (e)computing the average relevant document distribution over k queries bytaking an average of that number of relevant documents retrieved by theset of queries after each document retrieved; and (f) passing to amaximization procedure distributions and the total number of documentsto be retrieved once the average relevant document distribution iscomputed for the current query for each search engine, for finding thecut-off level for each search engine that maximizes the number ofrelevant documents retrieved, whereby said computed cut-off levelscorrespond to the number of documents selected from the result set ofeach search engine.
 15. A computer-implemented method for facilitatingWorld Wide Web Searches and like searches by combining search resultdocuments, as provided by separate search engines in response to aquery, into one single integrated list so as to produce a ranked list ofpages, said method comprising the steps of:forming a set of selectedqueries, said queries including respective terms, for which selectedqueries relevance data from past data is known, herein referred to astraining queries, in a vector space comprising all training queries,said relevance data comprising judgments by a user as to whether a pageis appropriate for a query which retrieved it, said forming a set ofselected queries including weighting each given term in said selectedqueries using a function that is proportional to that number of timessaid given term occurs in the query, creating a query vector for acurrent query in said vector space, computing a vector similaritymeasure between said current query and all training queries, selectingthose k training queries with the highest similarities, identifying aset of k most similar training queries to current query q, computing anaverage relevant document distribution of said k queries within saidtraining queries' search results for each of said search engines,wherein said average relevant document distribution is such that arelevant document distribution for a query q gives for each rank r thenumber of relevant documents retrieved at or below rank r by query q andthe average distribution over a set of queries gives the mean number ofrelevant documents retrieved at or below rank r over the query set,using said computed relevant document distributions, finding an optimalnumber of pages to select from the result set of each search engine whenN total pages are to be retrieved, computing, by using a selectedoptimization technique, a number of pages that should be retrieved fromeach respective search engine such that a total number of pages N isretrieved and a maximum possible number of relevant pages is retrievedsubject to a predetermined constraint, said predetermined constraint istypified by a constraint such that to retrieve a page at rank R from acollection, pages at rank 1 through rank (R-1) must also be retrieved,wherein, if in step (D) there be different combinations of pagesretrieved from those search engine results that retrieve a maximumpossible number of relevant pages, then select any one of said differentcombinations, distributing a number of pages that can be retrieved fromany of said search engines without affecting the number of relevantdocuments retrieved, in proportion to that number of pages that wouldotherwise be retrieved from that collection, creating a final retrievedset by forming the union of the top λ_(s) pages from each search engine,ranking pages in said final retrieved set probabilistically inaccordance with a rule of a die roll using a biased c-faced die,wherein, to select the page to be in the next rank r of the finalranking, rolling a c-faced die that is biased by the number of pagesremaining to be placed in said final ranking from each of said searchengines, selecting that search engine whose number corresponds to saiddie roll; and placing the next page from that engine's ranking into saidfinal ranking, and repeating said steps of selecting that search enginewhose number corresponds to said die roll and placing the next page fromthat engine's ranking into said final ranking are repeated until all Npages have been placed in said final ranking.